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(54) Movement vector detecting device 

(57) There is disclosed a movement vector detecting 
device provided with a movement vector detecting circuit 
for detecting a movement vector in each of blocks 
obtained by dividing an image frame, a detecting circuit 
for detecting the number of inversions with respect to a 
predetermined threshold value in each of the blocks, and 
operation means for varying the weighting of the move- 

FIG. 



ment vector value detected by said movement vector 
detecting means for each block, based on the output of 
said detecting means for each block, based on the output 
of said detection means, and effecting a predetermined 
averaging operation, thereby calculating a movement 
vector for the entire image frame. 
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Description 

BACKGROUN n OP THE INVENTION 
Field of the Invention 

The present invention relates to a movement vector 
detecting device, and more particularly to a device for 
detecting a movement vector from an image signal. 

Related Background Art 



The movement vector detecting device has been 
employed in the image encoding device or the image 
vibration compensating device. For movement vector is 
detection by image signal processing, there have been 
known a time-space inclination method disclosed for 
example in the U.S. Patent No. 3,890,462 and in the Jap- 
anese Patent Publication No. 60-46878, a correlation 
method based on calculation of correlation, and a block 20 
matching method. 

In the time-space inclination method, the amount of 
movement is calculated from the difference d in lumi- 
nance between frames (or fields) and the difference A 
between pixels in a frame. There is utilized a property 25 
that the image signal of a moving image is an average 
in time with afield cycle time, and that the edge becomes 
less sharp and the difference A in luminance among pix- 
els becomes smaller as the amount of movement of 
image becomes larger. The amount of movement is 30 
defined by d/A, namely the difference d in luminance 
among frames or fields, normalized by the difference A 
in luminance among pixels. Thedetails of the time-space 
inclination method are described by B. K. P. Horn et al., 
Artificial Intelligence 17, p.185 - 203 (1981). . 35 

The block matching method consists of dividing the 
input image signal into blocks of a suitable size (for exam- 
ple 8 pixels by 8 lines), comparing each block with the 
pixels of a predetermined area in a preceding frame (or 
field) and determining the most resembling position by 40 
laterally moving the comparing position within the image 
frame. For example there is searched a position where 
the sum, within the block, of absolute difference of pixels 
between the frames (or fields) , and the movement vector 
is represented by the relative displacement of the most 45 
resembling blocks. The details of the block matching cal- 
culation are reported by M. Ogami et al., Information 
Processing, Vol. 17, No. 7, p.634 - 640 July, 1976. 

In these methods, however, it has been difficult to 
exactly detect the movement vector, or the detected so 
movement vector has not been reliable, in case the unit 
block for movement vector detection contains only low 
spatial frequencies and lacks characteristic pattern 
(such as sky, water surface, white wall or asphalt sur- 
face), or in case said unit block contains a plurality of ss 
similar characteristic points with a high spatial frequency 
(such as flower field, leaves of a tree or a door with grid 
pattern), or in case an object having the edge only in a 



specified direction (such as an oblong rod) moves along 
the direction of said edge. 

W a high-efficiency image encoding device or an 
image vibration compensating device is operated 
5 according to an erroneous movement vector or a move- 
ment vector involving a large error, the precision of 
encoding or compensation is significantly deteriorated, 
and the image quality may become even worse by such 
encoding or compensation. 
10 In the field of vibration detection and vibration com- 
pensation utilising the movement vector, the present 
applicant already has the following granted US patents: 
U.S. Patents Nos. 5,198.896, 5.189,513. 
5,296,925, 4,965,892, 5,012,270 and 5,107,293. 

The present invention is to resolve the above-men- 
tioned drawbacks of the prior art, and is concerned with 
providing a movement vector detecting device capable 
of accurately detecting the movement regardless of the 
state of the image. 

The present invention is also concerned with provid- 
ing a vibration compensating device capable of accu- 
rately detecting the movement vector of an object image 
regardless of the pattern thereof, and enabling the real- 
isation of optimum movement compensating character- 
istics without erroneous operations. 

The present invention is additionally concerned with 
providing a camera constantly capable of accurate com- 
pensation for vibration regardless of the state cf the 
object to be photographed. 

In accordance with the present invention there is 
provided an image pick-up device as set out in claim 1. 

Still other features of the present invention will 
become fully apparent from the following description to 
be taken in conjunction with the attached drawings, 
which are given by way of example. 



RRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a block diagram of an embodiment of the 
present invention; 

Fig. 2 is a view showing the relation between the 
spatial frequency and binary patterns; 
Figs. 3A and 3B are views for explaining binary pat- 
terns; 

Fig. 4 is a chart showing an example of the evalua- 
tion function; 

Figs. 5A to 5C are charts showing the principle of 
edge detection by the zero crossing method; and 
Fig. 6 is a view showing a binary pattern having 
edges periodically on a vertical line. 



DFTAILED DESCRIPTION OF THF PREFERRED 
EMBODIMENTS 



Now the present invention will be clarified in detail 
by an embodiment thereof shown in the attached draw- 
ings. 

Fig. 1 is a block diagram of a video camera with an 
antivibration devices, constituting an embodiment of the 
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present invention. In this embodiment, the vibration of 
image is suppressed by storing the phototaken image 
signal of an image frame in an image memory, and con- 
trolling the read-out range from said image memory 
according to the detected movement vector. 

In Fig. 1, an image pickup device 12 converts an 
object image, formed by a phototaking tens 10, into an 
electrical signal. In the present embodiment the image 
pickup device 12 releases a line-sequential color differ- 
ence image signal. The output of said image pickup 
device 12 is supplied, through a sample hold (S/H) circuit 
14, to an AGC (automatic gain control) circuit 16. Which 
automatically regulates the gain of the output signal of 
the S/H circuit 14 and of which output is converted into 
a digital signal by an A/D converting circuit 18. 

A Y/C separation circuit 20 is composed of two 1 H 
delay lines and an adder tor adding the input and a 2H 
delayed signal. The output of said adder is supplied to a 
C process circuit 22 constituting a color signal process- 
ing circuit, while a 1 H delayed signal is supplied to a low- 
passf ilter (LPF) 28 of a luminance signal processing sys- 
tem. The C process circuit 22 generates a chroma signal 
from the output of the Y/C separation circuit 20, and its 
output is temporarily stored in a memory 23 in an image 
vibration suppressing circuit 33 to be explained later. 

In the luminance signal processing system, the LPF 
28 eliminates the carrier component of the line-sequen- 
tial color difference signal, from the signal supplied from 
the Y/C separation circuit 20. The output of the LPF 28 
is supplied to an enhancer 30 for edge enhancement. 
The enhancer 30 normally adds, to the input signal, a 
secondary differentiated signal thereof. A gamma cor- 
rection circuit 32 avoids the saturation in the high-light 
area and widens the dynamic range, on the output of the 
enhancer 30. The ordinary luminance signal process is 
completed in this manner, and the output of the gamma 
correction circuit is supplied to the image vibration sup- 
pressing circuit 33. 

In said image vibration suppressing circuit 33, the 
output of the gamma correction circuit 32 is supplied to 
a field (or frame) memory 34 and a band-pass filter (BPF) 
36 consisting of a spatial frequency filter. A memory 34 
temporarily stores the entered luminance signal. The 
BPF 36 extracts, from the oulput of the gamma correction 
circuit 32, spatial frequency components useful for 
movement vector detection. Thus the low and high spa- 
tial frequency components are eliminated from the imag e 
signal. 

In the present invention, the BPF 36 releases the 
code bit of the digital output signal of the gamma correc- 
tion circuit 32. This corresponds to the binarization of the 
luminance signal, employing the DC level as the thresh- 
old value. Thus, the circuits succeeding to the BPF 36 
can be composed of 1-bit processing system and can 
therefore be compactized. 

The output of the BPF is supplied to a movement 
vector detecting circuit 38, a memory 40 serving as 1- 
field time delay means and an edge counter 42. Said 
movement vector detecting circuit 38 also receives the 



signal of the preceding lield from the memory 40, and 
detects the movement vector by the calculation accord- 
ing to the time-space inclination method or by the corre- 
lation calculation, based on the current field and the 

5 preceding field. 

The edge counter 42 counts the number of edges in 
the horizontal and vertical directions contained in each 
unit block for movement vector detection in the move- 
ment vector detecting circuit 38. More specifically, it 

io detects the inversion of output of the BPF 36, by taking 
exclusive logic sum between the neighboring pixels. 

The movement vector (horizontal and vertical com- 
ponents) detected by the movement vector detecting cir- 
cuit 38, and the number of edges in each block, detected 

is by the edge counter 42, are supplied to an operation cir- 
cuit 44 composed of a microcomputer, which evaluates 
the reliability of the movement vector in each block, by 
the number of edges in the same block. More specifically, 
the movement vectors respectively detected in the 

20 blocks constituting the detection units for the movement 
vector are weighted by the number of detected edges, 
and the movement vector for the entire image frame is 
detected, as will be explained later in more details. 
The operation circuit 44 supplies the calculated 

25 movement vector to a memory read-out control circuit 
46, which, in response, controls read-out control circuit 
46, which, in response, controls the read -out positions 
of the memories 23, 34 so as to cancel the image move- 
ment. 

30 The chroma signal read from the memory 23 is con- 
verted into an analog signal by a D/A converter 24, and 
is released from an output terminal 26. Also the lumi- 
nance signal read from the memory 34 is converted into 
an analog signal by a D/A converter 48, and is released 

35 from an output terminal 50. In this manner, the image 
signal with suppressed image vibration is released from 
said output terminals. 26, 50. 

In the following there will be given a detailed expla- 
nation on the calculation in the operation circuit 44, 

40 namely the evaluation of reliability of the movement vec- 
tor, with reference to Figs. 2, 3A, 3B and 4. Fig. 2 shows 
an orthogonal coordinate system representing the spa- 
tial frequency area, wherein the abscissa fx indicates the 
horizontal spatial frequency while the ordinate indicates 

45 the vertical spatial frequency. Pattern shown on the coor- 
dinate system indicates a binary pattern corresponding 
to the spatial frequency of the illustrated position. In gen- 
eral, the spatial frequency distribution of an image 
assumes a cross-shape pattern with limited diagonal 

so components. 

In the movement vector in general, the component 
of movement in a direction orthogonal to the direction of 
edge (tangential direction) of the image can be detected 
with satisfactory precision, but the component in the 

55 direction of edge is difficult to detect, with low reliability 
of detected value. Also in the direction orthogonal to the 
edge, the reliability becomes low if the periodical pattern 
exists in said direction, because of the possibility of mis- 
matching. 
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For example, in case of an edge extending in the ver- 
tical (y) direction, the x component of the movement vec- 
tor is detectable, but y component is not detectable. In 
case of a rough pattern as shown in Fig. 3 A, with a block 
size of 10 x 10 pixels, there are observed 10 edges in 
the x-direction and 0 edges in the y-direction. In such 
case, the detected value of y-component of the move- 
ment vector is unreliable. 

Also in case of a pattern shown in Fig. 3B, with a 
higher spatial frequency with periodicity in the x-direc- 
tion, the detected value in the y-direction is unreliable 
because there is no edge in the y-direction as in the case 
of Fig. 3A. The reliability becomes also low in the x-direc- 
tion, because of the possible mismatching resulting from 
the periodicity. If a block size of 1 0 x 1 0 pixels is employed 
for the pattern of Fig. 3B, there are obtained 90 (= 9 x 
10) edges in the x-direction and 0 edge in the y-direction. 
The detected value in the x-direction is unreliable 
because of too many edges, and the detected value in 
the y-direction is also unreliable because of absence of 
edge. 

Fig. 4 shows an example of the weighting function 
(evaluating function) for the number of detected edges, 
wherein n indicates the block size in the x- and y-direc- 
tions. The maximum number of edges in a block of n x n 
pixels is n(n - 1). When the number of edges is close to 
zero, the evaluation is low because the effective informa- 
tion for movement vector detection is deficient. On the 
other hand, the evaluation is also low if the number of 
edges is very large, because mismatching can happen 
easily. The operation circuit 44 evaluates the x- and y- 
components of the detected movement vectors, individ- 
ually utilizing said weighting function, and effects a proc- 
ess of averaging similar movement vectors with weights 
in a closed output area and eliminating the movement 
vectors of low reliability. 

In the foregoing embodiment, the edge is judged 
from a particular spatial frequency component extracted 
by the two-dimensional BPF 36, but it is naturally possi- 
ble also to count the edges of boundaries or contours by 
extracting edge information from multi-value image sig- 
nal. Direct edge detection from the luminance signal can 
be achieved by the zero-crossing method, in which the 
edge is detected by secondary differentiation of the lumi- 
nance signal, as detaiiedly reported by D. Marr and E. 
Hildreth in "Theory of Edge Detection", Proceeding of the 
Roval Society of London, B207, 1980, pp187 - 217. 

Fig. 5A shows an edge part of an image, while Fig. 
5B shows a differential of the signal shown in Fig. 5A, 
and Fig. 5C shows a differential of the signal shown in 
Fig. 5B, or a second-order differential of the signal shown 
in Fig. 5 A. The wave form of said second-order differen- 
tial shown in Fig. 5C inverts the sign at the center of the 
edge, and said sign inverting point is called zero crossing 
point. Thus the edges can be detected in the horizontal 
and vertical directions, by effecting second-order differ- 
entiation in said directions. The reliability of the detected 
movement vector can be evaluated also by the edges 
detected by such zero crossing method. 



There have also been proposed other types of edge 
detectors, which are likewise applicable to the present 
invention. For example there are already known Roberts' 
edge detector reported by L. G. Roberts, "machine per- 
5 ception of three-dimensional solids", in Optical and Elec- 
tro-optical Information Processing, and J. T. Tippettetal. 
<eds,), MIT Press, Cambridge, Mass., 1965, pp.159 - 
197, and Prewitfs edge detector reported by J. M. S. Pre- 
witt, "object enhancement and extraction" in Picture 
10 Processing and Psychopictorics; and B. S. Lipkine and 
A. RosenfekJ (eds.), Academic Press, New York, 1970. 

In the foregoing embodiment, the reliability of the 
movement vector is evaluated by the total sum of the 
number of edges in the x- or y-direction, but the above- 
15 mentioned evaluation function may also be applied to a 
horizontal or vertical scanning line in each block. The 
meaning of such method will be explained with reference 
to Fig. 6. 

Fig. 6 shows an example of binary pattern, with a 
20 block size of 10 x 1 0 pixels, and the distributions of the 
number of edges in the x-and y-direction are shown by 
histograms illustrated outside of said pattern. The pat- 
tern shown in Fig. 6 has edges repeating in the y-direc- 
tion with a short cycle, so that it is difficult to exactly 
25 detect the y-component of the movement vector. How- 
ever, this pattern has 10 edges in the x-direction and 9 
edges in the y-direction, so that the reliability is high in 
the evaluation by the total sum of the number of edges. 
For such pattern, the aforementioned evaluation 
30 function may be applied to each horizontal or vertical 
scanning line. For a block size of n x n pixels, the number 
of edges present on a horizontal or vertical scanning line 
is between 0 and (n - 1). Therefore, the evaluation 
(weight) is made lower in case the number of detected 
35 edges is 0 or (n - 1) or close thereto. With this method, 
the y-component of the movement vector detected in this 
block is evaluated low. 

As will be easily understood from the forgoing 
description, the reliability of plural detected movement 
40 vector values is evaluated by the numbers of edges 
detected in respective blocks, so that a movement vector 
of high reliability and high precision can be obtained in 
the entire image area or in a suitable closed area. Con- 
sequently there can be obtained satisfactory perform- 
45 ance in the moving image encoding device or in the 
image vibration compensating device. 



Claims 



An image pickup device comprising: 

image pickup means (12) for effecting photo- 
electric conversion on an image, formed on an 
image taking plane of the pickup means to generate 
an image signal; 

block dividing means (44) for dividing the 
image taking plane of said image pickup means into 
plural blocks; and 

movement vector detecting means (38, 40) 
for detecting a movement vector from said image 
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signal, for each of said plural blocks, and character- 
ised by 

detection means (40) for detecting the 
number of inversions, with respect to a predeter- 
mined threshold value, of the image signal, in each 5 
of said plural blocks; 

operation means (44) for varying the weight- 
ing of the movement vector value detected by said 
movement vector detecting means for each block, 
based on the output of said detection means for jo 
each block, based on the output of said detection 
means, and effecting a predetermined averaging 
operation, thereby calculating a movement vector for 
the entire image frame; and 

compensating means (44, 46, 34) for com- 75 
pensating the movement of the image, according to 
the movement vector for the entire image frame, cal- 
culated by said operation means. 

A device according to claim 1, wherein said move- 20 
ment vector detecting means includes filter means 
for extracting frequency components suitable for the 
detection of said movement vector and eliminating 
unnecessary low and high frequency components. 

25 

A device according to claim 2, wherein said filter 
means is adapted to binary digitize the luminance 
signal, employing a predetermined DC level as the 
threshold value. 

30 

A device according to claim 3, wherein said edge 
detection means is adapted to count the number of 
edges, in each of said blocks, by counting the 
number of inversions of said binary digitized lumi- 
nance signal in the vertical and horizontal directions. 35 

A device according to claim 4, wherein said edge 
detection means is adapted to detect the edge in 
said image signal by second-order differentiation of 
said image signal, thereby detecting the number of 40 
edges in each of said blocks. 

A device according to any preceding claim, further 
comprising an image memory for storing the image 
signal, corresponding to an image frame, generated 45 
by said image pickup means; 

wherein said compensating means is 
adapted, in the read-out of image information from 
said image memory, to vary the read-out position 
from said image memory, in such a direction as to so 
cancel the movement vector of the entire image 
frame released from said operation means. 
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(54) Movement vector detecting device 

(57) There is disclosed a movement vector detecting 
device provided with a movement vector detecting circuit 
for detecting a movement vector in each of blocks 
obtained by dividing an image frame, a detecting circuit 
for detecting the number of inversions with respect to a 
predetermined threshold value in each of the blocks, and 
operation means for varying the weighting of the move- 



ment vector value detected by said movement vector 
detecting means for each block, based on the output of 
said detecting means for each block, based on the output 
of said detection means, and effecting a predetermined 
averaging operation, thereby calculating a movement 
vector for the entire image frame. 
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